Method and device for measuring similarity between images

ABSTRACT

A method is disclosed for measuring similarity between images, comprising the performance of the following steps for each of the images: segmentation of the image into segments, classification of the segments as a function of their orientation to give classes, calculation of a histogram of the number of segments as a function of class, calculation of a histogram of the number of pixels belonging to the segments of one and the same class as a function of class, and comparing, for each image, the histograms calculated above to their respective histograms calculated for each of the rest of the images to give a measurement of similarity.

This application claims the benefit, under 35 U.S.C. § 365 ofInternational Application PCT/FR03/50053, filed Sep. 12, 2003, which waspublished in accordance with PCT Article 21(2) on Apr. 8 2004 in Frenchand which claims the benefit of French patent application No. 0211950,filed Sep. 27, 2002.

The invention relates to a method and device for measuring similaritybetween images.

The field of application is that of the calculation of the similaritybetween images. This similarity may be used within the framework of asearch for images by similarity. It may also be used to estimateclusters of images that are close according to the similarity criterionused. It pertains for example to the construction of video summaries.This consists of a search for similar images in a base consisting ofimages of a video sequence. The aim is to reduce the number of theseimages by retaining only a representative for a cluster of images thatare decreed to be similar. It also pertains to indexation consisting inselecting index images on the basis of their similarity to other imagesor quite simply in the search for specific images in the video sequence,on the basis of enquiry images.

BACKGROUND OF THE INVENTION

Numerous studies are interested in the measurement of similarity betweenimages on the basis of attributes assigned to the images. A certainnumber of them are now defined in the MPEG-7 standard. The invention isinterested in particular in the so-called structural attributes. Forexample, in the latest version of the standard, an attribute of contourtype is defined. It is extracted locally from blocks of images on thebasis of Gabor filters.

The attributes known from the prior art or proposed in the MPEG-7standard and the algorithms for processing these attributes formeasuring similarity, do not however make it possible, for certain typesof sequences, to detect similar images in a satisfactory manner, in sofar as they do not take account either of the image in its entirety, orof the details in the image. With a view to clustering images extractedfrom a video, the similarity measurement must be able to convey the factthat several images correspond to one and the same scene, characterizedby a unity of place. An information item characterizing solely thedetails of the image is not sufficient to fulfil this task. On the otherhand an information item characterizing the image solely in its entiretymay prove to be unsuitable in the case of partial modification of thestaging of the scene, for example by modification of the background ofthe scene.

SUMMARY OF THE INVENTION

The aim of the invention is to alleviate the aforesaid drawbacks. Itssubject is a method of measuring similarity between images,characterized in that it performs, for each image, the following steps:

-   -   segmentation of the image,    -   classification of the segments as a function of their        orientation to give classes,    -   calculation of a histogram of the number of segments as a        function of class,    -   calculation of a histogram of the number of pixels belonging to        the segments of one and the same class as a function of class,    -   comparison of the histograms of each image to give a measurement        of similarity.

According to a particular implementation, the process is characterizedin that it also calculates a histogram corresponding to the distributionof the segments about the centre of gravity of each class.

According to a particular embodiment, to calculate this histogram, itperforms a calculation of the standard deviation of the distances fromthe middles of the segments of a class to the centre of gravity of theclass considered.

According to a particular implementation, the comparison of thehistograms consists of a subtraction of the ordinates, class by classand of a sum, over the set of classes, of the values obtained for eachclass.

According to a particular implementation, the histograms are codedaccording to the MPEG-7 standard.

The invention also relates to a method of clustering images of adatabase, characterized in that the clustering is performed as afunction of the measurements of similarity according to the methoddescribed above to give clusters of images.

The invention also relates to a method of creating video summaries,characterized in that it selects at least one of the images of at leastone cluster calculated according to the above method.

The invention also relates to a method of video indexation,characterized in that it selects at least one of the images of at leastone cluster calculated according to the above method, as indexationimage.

The invention also relates to a device for measuring similarity betweenimages, characterized in that it comprises a circuit for processing andfor calculation of histograms receiving digital data defining theseimages so as to perform, for each of them, the following operations:

-   -   segmentation of the image,    -   classification of the segments as a function of their        orientation to give classes,    -   calculation of a histogram of the number of segments as a        function of class,    -   calculation of a histogram of the number of pixels belonging to        the segments of one and the same class as a function of class,    -   comparison of the histograms of each image to give a measurement        of similarity.

The main advantage of the invention is the implementation of powerfulalgorithms for characterizing an image and measuring the similaritybetween images on the basis of these algorithms, by virtue of the jointuse of attributes based on orientation of contours making it possible tocharacterize the number of segments, their size and also theirdistribution as a function of orientation.

The global description of the image is obtained on the basis of thehistograms of the orientations of the segments in the image, hencewithout taking account of the position of these segments in the image,and the local description is obtained on the basis of a measure ofcentres of gravity of classes of segments, which corresponds to therelative positions of the various segments of one and the same class.

The use of structural attributes such as these makes it possible tocharacterize the image both locally and globally, thus improving theeffectiveness in the search for similarity of images. Specifically, thedimensions of the similar objects may be very different from one imageto another, from one shot of the sequence to another.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features and advantages of the invention will become clearlyapparent in the following description given by way of nonlimitingexample, and offered in conjunction with the appended figures whichrepresent:

FIG. 1, a flowchart of the method according to the invention,

FIG. 2, a method of polygonal approximation,

FIG. 3, an exemplary search for similarity.

DESCRIPTION OF PREFERRED EMBODIMENTS

The method carries out a segmentation of the image. The “object”approach does not however exist here. It involves determining segmentsin the image which are approximations of the actual contours in theimage, for example on the basis of measurements of gradients, withoutworrying about the fact that they do or do not belong to a particularobject in the image.

FIG. 1 represents a flowchart defining the main steps of the method ofcalculating histograms of attributes characterizing an image.

A first phase which is a preprocessing of the image consists of adetection of contours in the image so as to obtain a map of contours anda segmentation of these contours so as to obtain a map of segments. Thesecond phase carries out a calculation of the attributes for thedetection of similarity.

An image to be processed is transmitted to a first step referenced 1 inthe flowchart of FIG. 1. This step of the method carries out acalculation of gradients in the image. A gradients detection filter isapplied to the image to give at each point a measurement of gradient. Amap of vertical gradients and a map of horizontal gradients are thuscalculated. The norm of the gradients that is obtained on the basis ofthese maps, the square root of the sum of the squares of the horizontaland vertical gradient values attributed to a pixel, is also utilized.

Step 2 carries out a selection of the pixels on the basis of thegradient values calculated in the previous step, compared withthresholds. This selection is refined through that of the points ofgreatest contrast in a given neighbourhood window. The aim is to removethick contours by selecting, in the neighbourhood of a pixel, the pixelshaving the steepest gradients, by considering the values of thehorizontal and vertical gradients. This also involves favouring thepixels neighbouring a pixel already selected as a contour. The isolatedcontour points are eliminated. We obtain, at this stage, a binary map ofcontours, each pixel of the image is labelled as contour or otherwise.

A test of connectivity is performed on the binary map of contours so asto gather together the neighbouring pixels, so as to thus constitutecontour lines. To do this, a procedure for labelling as adjoiningcomponents is implemented in the next step 3.

Two pixels are declared adjoining if there exists an uninterrupted pathof contour pixels linking these two pixels. Contour pixels are thus“chained together” in such a way as to obtain a continuity line. Theadjoining pixels are labelled, each pixel touching another pixel beingmarked with the same label. A list of adjoining components is obtainedfor each label, this list being the coordinates of the various pixels ofa continuity line relating to a label.

To chain each contour together from one end to the other, it isnecessary to eliminate multiple points, that is to say the junctions ofcontour lines. An algorithm tests for the presence of junctions andeliminates them. This step is carried out by tracking contours over eachadjoining component identified. The traversal of the pixels belonging toa given adjoining component is done gradually. A combination of filtersmakes it possible to ascertain whether the current pixel corresponds toa junction between the main contour and a secondary contour. A pixelhaving been categorized as a junction is eliminated from the map ofcontour pixels, thus detaching the secondary contour from the maincontour. The secondary contour is replaced in the list of adjoiningcomponents to be traversed. The tracking of contours is then continuedalong the main contour.

At this stage, we have a map of nonrectilinear continuity lines.

The next step 4 of the preprocessing procedure is a polygonalapproximation of these chained contours so as to approximate eachcontour line by a set of connected segments.

FIG. 2 represents a method of polygonal approximation of a chainedcontour. For a given contour referenced 11, we search for the point e3giving the maximum distance between the points of the contour and thesegment formed by the ends of this contour e1 and e2. If this distanceis greater than a threshold, the contour is approximated by two segments[e1, e3] and [e3, e2]. The procedure is repeated until the distancesfrom the point to the segment are less than the fixed threshold.

On completion of this step, we have, for an image, a segment mapconsisting of a set of segments whose orientation can then becalculated. An example of a map of segments is represented at the topright of FIG. 3.

The second phase consists of a calculation and a utilization of theattributes relating to this segment map.

Step 5 calculates, for each segment of the map of segments, the anglebetween its direction and the horizontal. The angle obtained, lyingbetween 0 and 180 degrees, is then quantized to obtain a reduced numberof categories or classes of angles. For example, the angles obtained areclustered into 36 classes, the quantization interval being 5 degrees or,stated otherwise, a class corresponds to a width of 5 degrees.

Histograms relating to three attributes are now calculated during step6, utilizing the segment orientation information obtained previously.

-   -   a histogram of the number of segments as a function of        orientation.

The various classes are represented as abscissa and the occurrences asordinates.

-   -   a histogram of the total number of pixels belonging to segments        of like orientation as a function of their orientation.

The ordinates correspond to the sum of the lengths of the segments ofthe class, for a given class.

-   -   a histogram corresponding to the distribution of the segments        about the centre of gravity of each class, by calculating the        standard deviation of the distances from the middles of the        segments to the centre of gravity.

The centre of gravity G_(i) of class C_(i) is obtained by calculatingthe barycentre of the middles M(s_(j)) of the segments s_(j). The weightassociated with each middle is equal to the number of pixels n(s_(j))making up the segment:

$G_{i} = \frac{\sum\limits_{s_{j} \in C_{i}}{{n\left( s_{j} \right)}*{M\left( s_{j} \right)}}}{\sum\limits_{s_{j} \in C_{i}}{n\left( s_{j} \right)}}$

The ordinates H(i) correspond to the standard deviation of the Euclideandistances d(G_(i),M(s_(j))) between the barycentre or centre of gravityof the class C_(i) considered and the centre of each segment of theclass:

${H(i)} = \sqrt{\frac{{{{card}\left( C_{i} \right)}*{\sum\limits_{s_{j} \in C_{i}}{d\left( {G_{i},{M\left( s_{j} \right)}} \right)}^{2}}} - \left( {\sum\limits_{s_{j} \in C_{i}}{d\left( {G_{i},{M\left( s_{j} \right)}} \right)}} \right)^{2}}{{card}\left( C_{i} \right)*{card}\left( C_{i} \right)}}$where card is the cardinal function returning the number of segments ofthe class considered.

The distribution is put into the form of a histogram, that is to say ofvectors of values, thus allowing its utilization within the framework ofthe MPEG7 standard which defines the coding of such histograms.

In order to more easily combine the information, the histograms arenormalized, for example on the basis of the maximum values found.

The three histograms of the attributes characterizing the image areutilized for the similarity measurementments, step 7.

Three histograms are calculated for a first image which is for examplean enquiry image and three other histograms for a second image which isan image in a database. The measurement of similarity between the twoimages I₁ and I₂, that is to say the calculation of the distance betweenthese images may then be performed by simple subtraction of theordinates, class by class, for each type of histogram H^(k) of sizem_(k):

${d_{k}\left( {H_{1}^{k},H_{2}^{k}} \right)} = \frac{\sum\limits_{i}{{{H_{1}^{k}(i)} - {H_{2}^{k}(i)}}}}{m_{k}}$

A similarity indicator S(I₁, I₂) is for example the sum of the valuesobtained by these subtractions, for the whole set of histograms:

${S\left( {I_{1},I_{2}} \right)} = \frac{\sum\limits_{{k = 1},2,3}{d_{k}\left( {H_{1}^{k},H_{2}^{k}} \right)}}{3}$

The similarity measurement may be improved by comparing the class alsowith neighbouring classes of the histogram of the other image, with aweighting factor.

The histogram comparison uses for example a metric of cross quadratictype making it possible to circumvent the small variations in globalorientation between two successive images, for example during a smallrotation of the camera.

An exemplary search for similar images is given in FIG. 3, showing, atthe top left, the image considered, its associated map of contours andof segments, as well as the images recognized as the most similar in abase of 150 images arising from two different sequences.

The main characteristic of these histograms is therefore of providing adescription of the images which is both global and local. It is thuspossible, by virtue of the global description, to differentiate globallystructured images, for example images of towns characterized byorientations of horizontal and vertical segments, from image is ofcountryside characterized by orientations of segments that are morerandom. However, it is also possible, by virtue of the localdescription, to differentiate locally structured images, part of theimage is for example devoted to a building, another to a wood, from lessstructured images, for example a house in the middle of the forest.

The calculation of the segments, in the previous description, isperformed over the values of luminance of the image. It is of coursejust as conceivable to perform such a calculation for each of the colourcomponents of the image. If the calculations need more processingcapacity, on the other hand the production of histograms based on colourmakes it possible to improve the effectiveness of the method.

Three histograms are utilized here to characterize the image. It is ofcourse possible to utilize just one of them or a combination of two ofthem to calculate the similarity of images, without departing from thefield of the invention.

The segment orientations utilized may be the angles formed withreference straight lines other than the horizontal. The dimensions ofthe classes may be less than or greater than 5 degrees, depending on thecalculation power or time available or the quality of measurementdesired.

1. Method of measuring similarity between images, comprising:performing, for each of the images, the following steps: (a)segmentation of the image into segments, (b) classification of thesegments as a function of their orientation to give classes, (c)calculation of a histogram of the number of segments as a function ofclass, (d) calculation of a histogram of the number of pixels belongingto the segments of one and the same class as a function of class,comparing, for each image, the histograms calculated above to theirrespective histograms calculated for each of the rest of the images togive a measurement of similarity.
 2. Method according to claim 1, alsocalculating a third histogram corresponding to the distribution of thesegments about the centre of gravity of each class.
 3. Method accordingto claim 2, wherein, to calculate the third histogram, it performs acalculation of the standard deviation of the distances from the middlesof the segments of a class to the centre of gravity of the classconsidered.
 4. Method according to claim 1, characterized in thatwherein the comparison of the histograms consists of a subtraction ofthe ordinates class by class and of a sum, over the set of classes, ofthe subtraction result obtained for each class.
 5. Method according toclaim 1, wherein the histograms are coded according to the MPEG-7standard.
 6. Method of clustering images of a database, wherein theclustering is performed as a function of the measurements of similarityaccording to the method of claim 1 to give clusters of images.
 7. Methodof creating video summaries, implementing a selection of at least one ofthe images of at least one cluster calculated according to the method ofclaim
 6. 8. Method of video indexation, implementing a selection of atleast one of the images of at least one cluster calculated according tothe method of claim 6, as indexation image.
 9. Device for measuringsimilarity between images, comprising a circuit capable of receivingdigital data defining the images performing, for each of the images, thefollowing steps: (a) segmentation of the image into segments, (b)classification of the segments as a function of their orientation togive classes, (c) calculation of a histogram of the number of segmentsas a function of class, (d) calculation of a histogram of the number ofpixels belonging to the segments of one and the same class as a functionof class, comparing, for each image, the histograms calculated above totheir respective histograms calculated for each of the rest of theimages to give a measurement of similarity.